{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 对偶表示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "机器学习模型可以大致分为两类。\n",
    "\n",
    "一种类似于神经网络、线性回归等，我们选用一些数学函数，通过在训练集上迭代训练以调节参数，最终我们得到这个包含数学函数和参数的模型，预测新数据时只需将新数据传入模型。 \n",
    "\n",
    "另一种方法在对新数据做预测时需要训练数据。包含两种情况：\n",
    "1. 保留全部训练数据，如KNN和这章的Gaussian process。\n",
    "2. 保留部分训练数据，如SVM只需保存支持向量。\n",
    "每当预测新数据时，使用某种度量方法，用存储的训练集数据预测新数据。这种方法往往训练速度很快或者根本不需要训练，然而预测可能会比较慢。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于基于固定非线性特征空间映射$\\phi(x)$的模型来说，核函数形式为:\n",
    "$$k(x,x')=\\phi(x)^T\\phi(x')$$\n",
    "回归和分类的线性模型都可以用对偶表示重写，如一个线性模型，其参数通过最小正则化平方和误差函数来确定：\n",
    "$$J(w)=\\frac{1}{2}\\sum_{n=1}^{N}\\{w^T\\phi(x_n)-t_n\\}^2+\\frac{\\lambda}{2}w^Tw$$\n",
    "可以解出:\n",
    "$$w=-\\frac{1}{\\lambda}\\sum_{n=1}^N\\{w^T\\phi(x_n)-t_n\\}\\phi(x_n)=\\sum_{n=1}^Na_n\\phi(x_n)=\\Phi^Ta$$\n",
    "其中$\\Phi$的第n行为$\\phi(x_n)^T$，其中\n",
    "$$a_n=-\\frac{1}{\\lambda}\\{w^T\\phi(x_n)-t_n\\}$$\n",
    "将$w=\\Phi^Ta$代入$J(w)$可以得到：\n",
    "$$J(a)=\\frac{1}{2}a^T\\Phi\\Phi^T\\Phi\\Phi^Ta-a^T\\Phi\\Phi^Tt+\\frac{1}{2}t^Tt+\\frac{\\lambda}{2}a^T\\Phi\\Phi^Ta$$\n",
    "定义Gram矩阵$K=\\Phi\\Phi^T$,一个NxN的对称矩阵，原为:\n",
    "$$K_{nm}=\\phi(x_n)^T\\phi(x_m)=k(x_n,x_m)$$\n",
    "平方和误差函数可以写成:\n",
    "$$\n",
    "J(a)=\\frac{1}{2}a^TKKa-a^TKt+\\frac{1}{2}t^Tt+\\frac{\\lambda}{2}a^TKa\n",
    "$$\n",
    "再求解a:\n",
    "$$a=(K+\\lambda I_N)^{-1}t$$\n",
    "再代入线性模型，对于新的x，可以预测：\n",
    "$$y(x)=w^T\\phi(x)=a^T\\Phi\\phi(x)=k(x)^T(K+\\lambda I_N)^{-1}t$$\n",
    "对偶公式使得最⼩平⽅\n",
    "问题的解完全通过核函数$k(x; x′)$表示，因为a的解可以被表⽰为$\\phi(x)$的线性组合，从⽽我们可以使⽤参数向量$w$恢复出原始的公式。\n",
    "对偶公式的优点是，它可以完全通过核\n",
    "函数$k(x; x′)$来表⽰。于是，我们可以直接针对核函数进⾏计算，避免了显式地引⼊特征向\n",
    "量$\\phi(x)$，这使得我们可以隐式地使⽤⾼维特征空间，甚⾄⽆限维特征空间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(r\"../\")\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from prml.kernel import (\n",
    "    PolynomialKernel,\n",
    "    RBF,\n",
    "    GaussianProcessClassifier,\n",
    "    GaussianProcessRegressor\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_toy_data(func, n=10, std=1., domain=[0., 1.]):\n",
    "    x = np.linspace(domain[0], domain[1], n)\n",
    "    t = func(x) + np.random.normal(scale=std, size=n)\n",
    "    return x, t\n",
    "\n",
    "def sinusoidal(x):\n",
    "        return np.sin(2 * np.pi * x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8TecfwPHPk0QSJHZqz1Kz1RK0dlVRVZuiapYaraq9\nQuxd/VGK1labokXtHWqPqBXUqpFU7RhJnt8fT7RGQiI39yT3ft+v133Jvee553xP0p7vOc9UWmuE\nEEI4HxerAxBCCGENSQBCCOGkJAEIIYSTkgQghBBOShKAEEI4KUkAQgjhpCQBCCGEk5IEIIQQTkoS\ngBBCOCk3qwN4nnTp0ukcOXJYHYYQQiQae/fuDdFa+8SkbIJOADly5GDPnj1WhyGEEImGUupsTMtK\nFZAQQjgpSQBCCOGkJAEIIYSTkgQghBBOShKAEEI4KZskAKXUVKXUVaVUYDTblVJqrFIqSCl1SClV\nxBbHFUII8fJs9QQwHajynO0fAHkiX62B7210XCGEEC/JJuMAtNZblFI5nlOkBjBTm/UndyqlUiml\nMmqtL9ni+CL2Qh+Gcvzv4xwNPkrw3WBCH4ZyL+wenm6eZPTOSEavjLyW9jWypcyGUsrqcIUQ8cBe\nA8EyA+cfe38h8rNnEoBSqjXmKYFs2bLZJThn8CD8AdvPbWflyZWsClrFH8F/oHnxetAZvTLyTtZ3\neC/ne9QrUA+f5DEaYCiESAQS3EhgrfVkYDKAr6+vrFgfUw8fwqlTcOwYhIRAaCjcu0eIvsvqW/tY\nELyJfcluciWVG+VylqdegXrk98lP/nT5yeidkaRuSfF08yQ0LJRLty5x6fYlAq8GEnA+gIDzASw5\nuoQOqzpQ6dVKNH+zObXz18bVxdXqsxZCxIG9EsBFIOtj77NEfiZe1sOHsGMHrFoFv/0GgYEQFvZM\nsXTAJ5EvgIhXUuNSNAmU9YCab0D6fE+U93L3Ik/aPORJm4ey2cvSrlg7AA5fOcycw3OYEziH+ovq\nkydNHnqU7kHjNxrj7uoev+cqhIgXylTL22BHpg3gV611oSi2fQh8AVQFSgBjtdbFX7RPX19fLXMB\nPeXsWZg4EX780dzpu7lB6dLwzjuQPz9Xs6Rm5KlZzDy5ELdkXnxRqCXNM31IhlAXOHoU9u6F3btN\nwgB47TVo2BBatYLMmV94+PCIcJYeW8rgrYPZf3k/r6Z+lbEfjKVqnqrxfOJCiJhQSu3VWvvGqLDW\nOs4vYC6mPv8hpn6/JdAGaBO5XQHjgVPAYcA3JvstWrSoFpGOHdO6Th2tXVzMq0YNrRct0vrGDa21\n1uER4XrCrgnaa4iXdh/orr/+7WsdfCc4+v2dP6/1+PFaV6yotVJau7pqXbeu1tu3xyiciIgIveLE\nCp3vu3waf3SNuTX0mX/O2OBEhRBxAezRMb12x7SgFS9JAFrrq1e1bt/eXKC9vbXu0UPrs2efKHLq\n2in97vR3Nf7oijMr6lPXTsXuGKdOad2li9Zp0pj/JKpW1Xrfvhh99X7YfT1823CdfHBy7TXES888\nMDN2xxZC2JQkAEcxb57WqVObi3/btlpfufJMkSV/LNHeQ7y19xBv/cPeH3RERMTLH+/2ba2HDTPH\nBK0bNdL60qUYffXs9bO6zNQyGn/0J4s/0Tfu3Xhi+5o1WjdurHXNmlpPmqT1vXsvH6YQInqSABK7\na9fMxRe0LlFC6yNHnikSFh6me6ztofFHF5tcTP/5z5+2O/4//2jdu7fWHh5ap0xpqorCwl74tbDw\nMN1/U3/t0t9F5x6bWx8LPqa11nrwYK1z5TK7mTtX6/fe07pCBa3v37ddyEIIQxJAYnbggNbZspm7\n/gEDtH748JkiN+7d0JVmVdL4o1svb63vPYyn2+kTJ0wbAWhdsqTWp0/H6Gtb/tyifUb46FTDUun5\nu9fpVKmefJAID9e6TBmtf/opfsIWwpnFJgHIZHAJyfLlUKoUhIdDQAD4+ZlePo+5dOsS5aaXY8OZ\nDfzw0Q9M+mgSHm4e8RNPnjywZg3MmgVHjkDhwjB7Nujn9xwrk70Mu1rtIrN3ZhquqEyOOj+QIcN/\n211c4JNPYO3a+AlbCBEzkgASitGjoWZNyJ8fdu2C4s/2kj3x9wlKTi3Jyb9P8kvDX/isyGcv3G1Y\nGMyda3p6NmsGq1e/8Pr9JKWgcWM4eBDefBM+/RSaNIG7d5/7tRypchDQMoCiqSpxIGtrRm4f+cT2\nS5cgTZpYxCGEsL2YPipY8XKaKqB+/Uw1S926Wt+5E2WRwCuB2meEj/YZ4aN3XdgVo92Gh2tdu7bW\nb7+t9dSpWn/3ndZ58pjq/ZcSFqZ1//6m2+ibb2r954vbHe7ee6CTN22g8Uf33dBPR0RE6MBArV95\nRetDh14yDiFEtJA2gEQiIkJrPz/zZ2jePNqG1j+u/qFfGfmKzjgq478NqzGxcqXWb7zxZGNrSIjW\n6dLFuDo/aitWmMbhtGm13rDhhcUPHArTKT5trvFHZ27WVadOE6FnzYrD8YUQ0YpNApAqICv17QsD\nB0LLlmZkr+uzc+scDzlOhZkVUCg2NN1A3nR5Y7z7NWtMXbv7YzM1pE0LVavCunVxiLtqVVNN9cor\nUKmSaRd4jsKvu/L3tB+pnbUtF3OMpN28gTRuHIfjCyFsQhKAVf73Pxg0CD77DCZPNi2jTzl34xwV\nZlYgQkewoekG8qXLF8WOopcyJVy58uznV66YbXHy2mumobp0adMuMHz4cxsX3FxdWNj8O5oWbsrg\ngH78b+f/4hiAECLOYvqoYMXLYauAliwx9ei1akVb7fP33b91/u/y65RDU+pDl1+usjwoyFT37N//\n32e//KJ1+vTRNjXE3r17WjdoYKqxvvzSNDw8x8Pwh7rWvFoaf/T0/dNtFIQQ4hFiUQWU4KaDdng7\nd0KjRqaXz+zZUVb73Au7R415NTj1zylWN17N6+lff6lDvfoqjB8P770Hr78Od+7A5cuwdCkkSxbX\nE4nk4QE//QSZMsE335hpqCdOjPK8ANxc3JhbZy7V5laj5fKWZE6RmYq5KtooGCFEbNhsNtD44HCz\ngZ4/D0WLgre3mcr5lVeeKRKhI6i/sD6Ljy5mft351C9YP86HvXMHtm411+oyZZ4ZWmAbWkO/fqZN\n49NPYerU5x7o5v2blJ5amnM3zrGj5Q7y++SPh6CEcD6xmQ1U2gDs5cEDqF/f3CGvWBHlxR/Af5M/\ni48uZnSl0Ta5+AMkTw5VqsC778bTxR/MeIEBA0y7xqxZZuxAFOsTPJLCIwW/NvoVTzdPPpzzIcF3\nguMpMCFEdCQB2EvXrqb6Z+pUyBd1Y+7CIwsZuGUgLd5swddvf23nAG2kd28YMQLmzzcN3BER0RbN\nljIbyxsu59LtS9SaX4sH4Q/sGKgQQhKAPcyfD2PHwldfQb16URY5ePkgzZY1450s7zDhwwmJeyH2\nrl2hf3+YMQO++OK5vYOKZy7OjJoz2H5+O51Wd7JjkEIIaQSOb6dPmzvhd94xd8ZRuBZ6jZrza5La\nMzWL6y+Ov7l97MnPD27fhpEjwcsr2nMHqF+wPrsu7mL0jtGUyFyCTwt/asdAhXBekgDiU3i4mYDH\nxQXmzXtyRFYkrTXNljbj4s2LbG2+lYzeGe0fZ3xQyowNuHPHJIGMGeHr6Ku1hlUcxt5Le/n81895\nI/0bFM5Q2I7BCuGcpAooPn37rel+M3YsZMsWZZFvdnzDLyd+YeT7IymRpYSdA4xnSplzr1sXOnUy\nSTAabi5uzKszj9RJU1NnQR1u3r9px0CFcE6SAOLLH3+YBtHq1c3smVHYcX4HPdb3oHb+2nQo0cHO\nAdqJq6vpFVSmDDRtChs3Rls0vVd6FtRdwJ/X/6TtirYk5C7KQjgCSQDxISzMXOy8vc00D1E06P4T\n+g8fL/qYrCmyMqX6lMTd6Psinp6wbJlZX6BWLTh6NNqipbKVwr+8P3MOz2HmwZl2DFII5yMJID6M\nGwd79sCECZA+/TObtda0XdGWS7cvMb/ufFJ5prIgSDtLnRpWrjSj0apVg5CQaIv2LN2TctnL0X5l\ne078fcKOQQrhXCQB2NqFC2aWz6pVTd13FOYcnsP8I/PxL+dPsczF7ByghbJlM08CFy9C7dpw/36U\nxVxdXJldezYebh40XNxQxgcIEU8kAdja11+bKqBx40Apbt6EbdsgKMhsPnv9LO1WtqNU1lL0KN3D\n2lit8PbbMG2aaRxv2zbaMQJZUmRhSvUp7Lu0j0FbBtk5SCGcgyQAW/rtN1i0CPr0gVy5GD0asmeH\nLl3MrMmVq0TQcGETtNbMqjULV5eoJ0xzeA0bmnEC06aZ2eqiUTNfTZoWbsqQrUPYfXG3HQMUwjlI\nArCVe/egfXvImxe6dGHZMpg0CQ4cMDNAnD8P9wqPY8dfWxj7wVhyps5pdcTW8veHjz4yT0xbtkRb\n7Nsq35LROyNNljYh9GGo/eITwglIArCVcePMqN/vvgMPDyZPNte47NnN5rO3gtidoidJznzI+z5N\nLQ01QXBxMd1Dc+Uy02OcPx9lsVSeqZhWYxrHQo7Re0NvOwcphGOTBGALISEweLDp3VLRzG0fHPzf\nxT9CR9ByeUvcXd3JdmgS1645cJfP2EiZ0ixOcPcu1KkTbaNwxVwVaefbjm93fkvA+QA7BymE45IE\nYAsDBph5bx6b7+bdd2HuXPPzhN0T2HJ2Cx3zj+HOpczkjfmyvo4vf36YORN274bOnaMtNvz94WRN\nmZXPln/G/bCoE4UQInYkAcTViRPw/ffQqpW5mEXq1Ml0e2/Y9k+6ru5B/iRVmNyuGSNHRjklkHOr\nVcv8wsaPNzOnRsHL3YtJ1SZxNOQoQ7YOsXOAQjgmSQBx1aOHGenq7//Ex+nTw86dmj3p2/PwIeQ7\nOYmFCxSNG1sTZoI3bBiULGlmTj1+PMoiVXJXofEbjRm6bSiBVwPtHKAQjkcSQFzs3Ak//wzdu0c5\n4ndz8CKC1EpGfjCQJdOyUaqUBTEmFkmSmLt/T08zgO7u3SiLjak8hpSeKWm5vCXhEeF2DlIIxyIJ\nIC769QMfH+jY8ZlN1+9dp8NvHSiSsQhflvjSguASoSxZYPZsCAyMtj0gXbJ0fFv5W3Zd3MXkvZPt\nHKAQjkUSwMvatg3WrIFu3cyCJ0/pua4nV+9cZXK1ybi5yLILMVa5sllRbOJEWLw4yiKNXm/Eeznf\no+f6nly5fcXOAQrhOCQBvKx+/Uy1T7t2z2z6/cLvTNo7iQ7FO1A0U1ELgkvkBg2CYsVMe8DZs89s\nVkoxvup4QsNC6bK2iwUBCuEYJAG8jM2bYcMGU/efLNkTm8Ijwmm/sj0ZvTMy4N0BFgWYyLm7m8Vj\nwsPhk0/M3EpPyZsuL91KdmP2odlsPBP9GgNCiOhJAogtrc1snxkyQJs2z2z+cd+P7L20l1Hvj8Lb\nw9uCAB1ErlymGmj7dtNDKAq9yvQiV+pctFvZTmYMFeIlSAKIra1bzdw1PXtC0qRPbAq5G0KvDb0o\nl70cDQo1sChAB9KokXn5+8OuXc9sTpokKeM+GMexkGOM+32c/eMTIpGTBBBbI0ZAunSmfvopvdb3\n4sa9G4yvOt6xV/iyp/HjIVMmaNzYjLZ+StU8Vfkwz4f039yfy7cvWxCgEImXJIDYCAyEFSugQ4dn\n6v73/rWXH/f9yFclvqLgKwUtCtABpUplpooICjKjhaMwpvIY7offp+f6nnYOTojEzSYJQClVRSl1\nXCkVpJR6ZpUTpVR5pdQNpdSByFdfWxzX7kaMgOTJzbTPj9Fa89VvX+GT3Id+5ftZFJwDK1/edA39\n4Qczv8ZT8qTNQ6e3OzH9wHR2Xthp//iESKTinACUUq7AeOADoADQUClVIIqiW7XWb0a+El/3mHPn\nzOxurVpBmjRPbFpwZAHbz29ncIXBpPBIYVGADm7AAChUyFS9Xbv2zObeZXuTyTsTX676kggdYUGA\nQiQ+tngCKA4Eaa1Pa60fAPOAGjbYb8IyZoz59+uvn/j47sO7dF3blTczvEnzN5tbEJiT8PAwVUHB\nwfDFF89s9nL3YnjF4ez5aw+zD822IEAhEh9bJIDMwOOreVyI/OxpJZVSh5RSq5RS0VaSK6VaK6X2\nKKX2BAcH2yA8G7h2zVQ/NGxoFjZ/zKiAUZy/eZ7/Vfmf8y7xaC9vvWUG4M2dCwsXPrO50euNKJap\nGD3X9+TOgzsWBChE4mKvRuB9QDat9RvAOGBpdAW11pO11r5aa18fHx87hfcCU6bAnTtmcd/HXLx5\nkeHbh1O3QF3KZi9rUXBOpkcPM0q4bVu48uQ0EC7KhTGVx/DXrb8YGTDSogCFSDxskQAuAlkfe58l\n8rN/aa1vaq1vR/68EkiilEpng2PHv/Bw0xWxfHl4440nNvXZ2IewiDBGVBwR9XeF7bm5wYwZpkvo\nU43xAKWylaJegXqM2D6CizcvRrEDIcQjtkgAu4E8SqmcSil3oAGw/PECSqkMKrJjvFKqeORx/7bB\nsePfL7+Y+Wi+fHJGzwOXDzDjwAw6FO8gC7zbW/78ZnDY4sVRVgUNrziccB1Orw297B+bEIlInBOA\n1joM+AJYDRwFFmitjyil2iilHs2VUBcIVEodBMYCDbTWOq7HtouxY029f/Xq/36ktabLmi6kTpqa\nXmXkImOJLl3A19dMxvdUW1HO1DnpWKIjMw/OZN+lfRYFKETCZ5M2AK31Sq31a1rrV7XWgyM/m6i1\nnhj583da64Ja68Ja67e11oljZe/AQNi40Vxk3P6b0vm3oN9Yf2Y9fcv2JXXS1BYG6MTc3GDaNLhx\n45mnMzDzBKVNmpZua7uRWO41hLA3GQn8POPGmRWqHpv2ISwijK5ru5I7TW7aFmtrYXCCQoVMr6D5\n82HZsic2pfRMiV9ZP9afWc+aU2ssClCIhE0SQHT++QdmzTLTEadN++/HMw/O5EjwEYa9Nwx3V1nd\n3XLdukHhwuYp7caNJza1LdaWXKlz0W1dN1k+UogoSAKIzqxZEBr6RE+T0Ieh9N3YlxKZS1A7f20L\ngxP/SpIEfvwRLl82yeAx7q7uDKkwhENXDsngMCGiIAkgKlqbgV++vmbwUaRxu8Zx8dZFhlccLrN9\nJiS+vmaiuMmTzWI9j6lXsB7FMhWjz8Y+hD4MtShAIRImSQBR+f130wDcuvW/H10LvcbQbUP5MM+H\nlMtRzsLgRJT694dXXzXtNaH/XehdlAsj3h/BhZsXmLB7AjduwMmT8EDWjxFCEkCUJk82s342+G9R\nl2HbhnHj3g2GvjfUwsBEtJIlM3+3oCAYPPiJTeVzlOf9nJXpvXoI2fLcoHJlyJrVLDgmhDOTBPC0\nmzdNr5KGDcHbLOl44eYFxv4+liaFm/B6+tctDlBEq0IFaNoUhg83T3CPSbV3CPddrtFq2ihOnzZL\nOo8aBUujnZRECMcnCeBpc+bA3btPVP8M2DwAjaZ/+f4WBiZiZNQos4hMq1YQYaaFvnUL1swoQs3c\nH/P9wW+4cvsKBQuapYbHyUqSwolJAnja5MmmW6GvLwAn/j7B1P1TaVO0DdlTZbc4OPFC6dLBN9/A\nzp3/1vFcuwZeXjCiykAehD9g0JZBABQoABcuWBmsENaSBPC4fftg/35z9xjZy6ffpn54unnKlA+J\nSePGULGimTn0r7/IHDk5+a2zefjsrc+YtHcSZ/45w/Ll8Pbb1oYqhJUkATxu5kxwd4dGjQAz4du8\nwHl0fLsj6b3SWxyciDGl4PvvTVefjh1xczPtwrVqQcG//XDBlTrj+jNmjMkRQjgrSQCPPHxo6v+r\nV4fUZn6fPhv6kNozNV1KdnnBl0WCkzs39O5tZgtdtYqmTWHSJFg5PxNex9pzIGIW0345Sv78Vgcq\nhHUkATyyerWZVbJJEwACzgew4uQKupXqRirPVBYHJ15Kt26QN68ZzX33LlWqmDXlj/3Qg+QeyZh+\ntp/VEQphKUkAj8ycCT4+UKUKAH4b/UifPD1fFn92pkmRSHh4mKqgM2eeGBuQLlk6vn77axb+sZD9\nl/ZbGKAQ1pIEAGbit2XLTN1/kiRsOLOBDWc20LN0T5K7J7c6OhEX775rnupGjoSjR//9uPM7nUnt\nmRq/jX4WBieEtSQBACxYYBoMmzRBa43fRj8ye2fmc9/PrY5M2MLIkWZkd/v2Zp4nzHTR3Up1Y8XJ\nFew4v8PiAIWwhiQAMNU/BQvCW2+x+tRqAs4H4FfWD083T6sjE7bwyiswdKhZ3Gfu3H8//qL4F/gk\n86Hvpr4WBieEdSQBBAVBQAA0bYrG9PzJkSoHzd9qbnVkwpZatYJixcysoZHrBni5e9GjdA/WnV7H\nlrNbLA5QCPuTBDBvnvm3YUOWH1/O3kt76Vu2ryz24mhcXU2DcHAw+P1X79/Gtw0ZvDLQd2NfWTpS\nOB1JAPPnQ5kyRGTORL9N/cidJjefFv7U6qhEfCha1KwcNn68GfENJEuSjF6le7H57GY2/rnR4gCF\nsC/nTgCBgebVoAFLjy3l4JWD9CvXDzcXtxd/VyROAwea+YLat/93srhWRVuRJUUW/Db6yVOAcCrO\nnQDmzwcXFyLq1Kbfpn7kTZuXhoUaWh2ViE+pUsGIEbBjB8yYAYCnmye9y/Qm4HyALCAvnIrzJgCt\nTf1/hQosDtlK4NVA+pXrh6uLq9WRifj26adQqpQZKfzPPwC0eKsFWVNkxX+zvzwFCKfhvAlg3z4I\nCiKifn38N/uTP11+6hesb3VUwh5cXEw7wLVr0KcPYBaQ712mNzsv7GT1qdUWByiEfThvApg3D5Ik\nYVkBF/4I/kPu/p1N4cKmHeD7783NAND8reZkT5kd/03yFCCcg3MmgIgImD8fXakSfQ5+QwGfAtQr\nWM/qqIS9DRhg5n/64guIiPj3KeD3i7/zW9BvVkcnRLxzzgSwcyecP8/vpXL8e/fvopzzV+HUUqUy\n6wfv2GFGgwNN32xqngKkLUA4Aee86i1ejHZ3p4P7Ogr4FKBugbpWRySs0qSJWRase3e4fh13V3f6\nlO3Drou7WBW0yurohIhXzpcAtIYlS7hUoiC7bx+Xu39n5+IC331nRgj7+wPQpHATcqTKQf/N/eUp\nQDg057vyHTgAf/7J99kuy92/MIoWhc8/N4ng8GHcXd3pVboXuy7ukh5BwqE5XwJYsoQIF8X3mS7R\nt2xfufsXxqBBkDIlfPklaP1fW4D0CBIOzOmufnrxYnbnTkr6HNLzRzwmbVqTBDZvhgULzFNAmV78\nfvF3GR0sHJZzJYCjR1FHjzLr1bv4lfWTu3/xpNat4c03oUsXuHOHZm82I1vKbNIjSDgsp7oCRixZ\nAsDhUq9Sr4Dc/YunuLqadoALF2DIkH/bAnZe2Mna02utjk4Im3OqBHBj7lR2ZIHPqw+QUb8iaqVK\nQePGMGoUBAXR7M1mZE2RVXoECYfksAkgPBxWr4axY2HDBgg/c4bUR06ztWg6Pi74sdXhiYRsxAhw\nd4dOnfBw86BH6R4EnA9gw5kNVkcmhE05ZAIICYESJaB3bzhxAjp2hKE1hgDwWsvucvcvni9jRrNq\n2C+/wKpVtHyrJZm9MzNgywCrIxPCplRCfqz19fXVe/bsifX3mjUzo/zHjAGlIDxcszV3SjLeuser\nV+/Kgi/ixR48gNdfNz8fPsy4/ZPo8FsHNjbdSPkc5S0NTTige/fg2DEzSaFScdqVUmqv1to3JmUd\n7gkgIgIWLDA3cI9+j2sOLKDkuVus9HxfLv4iZtzd4dtvzSPk2LG0KtqKjF4ZGbBZngKEbT18CNsH\nb4K33mLnwLWEh9vv2DZJAEqpKkqp40qpIKVUjyi2K6XU2Mjth5RSRWxx3KhoDWFh4OHx6L1m8+Re\nuEfAbze7xtdhhSP64AOoVg3698cz+B+6lerGxj83svXsVqsjEw7i8mV46y24NGUlD9yS0v3XMhQv\nDn//bZ/jxzkBKKVcgfHAB0ABoKFSqsBTxT4A8kS+WgPfx/W40XF1Nf/Pjh1r3q8KWkWBnae5kdSL\nzLXKxNdhhaMaM8ZUB/XsSeuirUmfPL20BQib6dIFqn6gqZt0Be6VK7Dp96S8/bZpv7QHWzwBFAeC\ntNantdYPgHlAjafK1ABmamMnkEopldEGx47S6NEweTLUrKVp91N/qp50YYNbDQYPk8ZfEUu5c0On\nTjBjBsn2HqJrya6sO72OgPMBVkcmEjmtYfFi6FXvJJw+TUj54igFPXqYamx7sEUCyAycf+z9hcjP\nYlvGZnLmhMBAyFFhDVnO7SJdaARVJ9QgY7ylHOHQeveGTJngyy9pU6Q1Psl8pC1AxJnWps3SY/1K\nAIpf6MfaU2txdTXb7CHBNQIrpVorpfYopfYEBwe/9H68vKB3iyKMuFcanSQJHtUr2zBK4VS8vMzY\ngD17SD5nIV1KdmH1qdX8fuF3qyMTiZiLC9SsCRd/WMmZjEl5mD0LZbKVZdQoqF3bTjHYYB8XgayP\nvc8S+VlsywCgtZ6stfbVWvv6+PjEKTCf5D6U3B+CKl8eUqSI076Ek2vUCEqWhB49aJenEWmTpmXg\nloFWRyUSuW8G3Cbrn5tYnCOUN2705INKHqxZA0OH2uf4tkgAu4E8SqmcSil3oAGw/Kkyy4Emkb2B\n3gZuaK0v2eDYzxcUZPrWfvRRvB9KODilYNw4CAnBa9g3dHqnEytOrmDPX7EfpyLEI5mPb8BDP2Rr\nvrQUjmhB27awbx+88op9jh/nBKC1DgO+AFYDR4EFWusjSqk2Sqk2kcVWAqeBIOAHoF1cjxsjFy6Y\nBgFJAMIWihSBzz6DcePo4F2R1J6p5SlAxMlf83/kljtUatGbIQM8qV/fDEGxF4ccCfwEreM8sk6I\nfwUHQ548UKwYA3uXoe/mfuxrvY+3Mr5ldWQisdGaK+mSsjcTvLvvH5ImSWqT3Tr1SOBnyMVf2JKP\nDwwYAOvW0enqq6T0SCnjAsRL2bduFumv3SdJtRo2u/jHluMnACFsrW1bKFiQ5D386FLkC5YeW8rB\nywetjkokMvt/MDcOpT8fbFkMkgCEiK0kScxQ8zNn6BwAKTxSSFuAiJWA8wEUDDjF5XxZSJojt2Vx\nSAIQ4mVow+h1AAAcb0lEQVRUqAB16pB05Bj65GjK4qOLOXzlsNVRiURi7LJevH0R0jRobmkckgCE\neFmjR0NEBB0WX8Db3VueAkSM7LywE+81mwFwr23t0rSSAIR4WdmzQ48eeCz6mW88a7Doj0UcuXrE\n6qhEAtd/c3/qnUxCRPbsUKiQpbFIAhAiLrp1g+zZaTZ1Pylck8lTgHiuXRd3seWP36hwWuNSs6bl\nvRQlAQgRF0mTwujRuAUeYebVUiw4soA/gv+wOiqRQPXf3J86571wexAGNZ6eNNn+JAEIEVe1a0OF\nClSbvYtsD+UpQERt18VdrDy5ki7Br5k1a0uXtjokSQBCxJlSMHYsLjdvMf/ga8wPnM/R4KNWRyUS\nmP6b+5POIzWFdv8JH35ouhNbTBKAELZQsCB88QXFVxygZLCnjA4WT3h09/+NV11c/r6WIKp/QBKA\nELbj749Kl46fNqVh/uF50iNI/Kv/5v6kSZqGj4+7mQXLq1SxOiRAEoAQtpMqFQwbRvYjF/nsiIc8\nBQjgsbr/Ep1wX/qLufh7e1sdFiAJQAjbatYMihVj9Ho3ftu7gMCrgVZHJCzmv8mftEnT0kG9baao\nr1vX6pD+JQlACFtycYHx4/G6fpdB25LI2sFObsf5HawKWkXXkl1JvnyVafhNQOuTSAIQwtaKFUO1\nbEm7HeEEbloocwQ5Mf/N/qRLlo72xdrBokXw/vuQMqXVYf1LEoAQ8WHIEJS3N+NXu+K/qZ/V0QgL\nbDu3jTWn1tCtZDe8Dh+Hs2cTVPUPSAIQIn74+OAyaDDvngrHZfHP7L+03+qIhJ3129SPV5K/QrtH\nd/9ubgmm++cjkgCEiC+ff074G4X4do1i6G+9rY5G2NHmPzez4cwGupfqTvIkyWDxYjOFeJo0Vof2\nBEkAQsQXNzdcJ0wk8w3NW1NXseviLqsjEnagtcZvox8ZvDLQxrcNHDoEQUEJrvoHJAEIEb9KleJh\n40Z03gGT53S2OhphB+tOr2Prua30LtObZEmSwbx54OoKNWtaHdozJAEIEc+SjPqGiKSe1J+4je1n\nt1kdjohHWmv6bOxD1hRZaVWkFUREwNy5UKkS+PhYHd4zJAEIEd/Sp0cNGEil07Bm5Odora2OSMST\nFSdXsOviLvqW64uHmwfs2GF6/zRqZHVoUZIEIIQdeHzZkeA8mWg18w82Bf5qdTgiHkToCPw2+pEr\ndS6aFm5qPpwzBzw9E1zvn0ckAQhhD25upJjyE1luwYWureUpwAEtObqEA5cP4F/OnySuSeDhQ1iw\nAKpXTzBz/zxNEoAQduJRpjzHq5eiwdrLbFox3upwhA2FRYTht9GP/Ony0+j1yOqe9eshJCTBVv+A\nJAAh7CrX5IXc9nTBq1MPIiLCrQ5H2MjsQ7M5FnKMQRUG4eriaj6cM8fMEJtApn6OiiQAIewoSfqM\nnOjcjGIn77BreAerwxE2cD/sPv029cM3ky+18tUyH969Cz//bPr+e3hYG+BzSAIQwo62bgW/HZP4\nPX1Scg2axKKpV60OScTR5L2TOXfjHEMqDEEpZT785Re4fRsaNrQ2uBeQBCCEnWzbBnXqwKdN3Age\nOYy0oeHc8q/HlClWRyZe1p0Hdxi0dRDlc5SnYq6K/22YPh2yZoVy5SyLLSYkAQhhJ4MHw8iR8Omn\n8GHjL1lYMRNNL2xhmd8mIiKsjk68jG93fsvVO1efvPu/eBHWrIGmTc0I4ARMEoAQdnLgALz3nvlZ\nKUX2b6fxlxcMvNWI6yFh1gYnYi3kbgjDtw+nZr6avJP1nf82zJplRgA3bWpdcDEkCUAIO8mdG/bu\n/e/9OwUq8cMnRSl8+xLuU4ZZF5h4KUO2DuHOwzsMqTDkvw+1NtU/pUubP3gCJwlACDvp3Bm++srM\nDqA1nDgBP5+cwq95IMmAAXDunNUhihj68/qfjN89nhZvtiC/T/7/NuzcCcePQ/Pm1gUXC5IAhLCT\nmjVh0CBo3BhSpIAyZaBBhcJs7FyHsPCHhLZuYTKDSPD6buyLi3LBv7z/kxumT4dkyaBePSvCijVJ\nAELYUePGZmr4ixfhr7+gVy/o2GAMA95zJenq9bBkidUhihc4ePkgsw/NpmOJjmROkfm/DXfvmqmf\n69ZNsFM/PE0SgBB2ppR5AnjUQSRryqy4duzE/gzwsH0buHHD2gDFc3Vb143USVPTvXT3Jzf8/DPc\nvJkoGn8fkQQgRALQrVwvutZNgcvVEOjZ0+pwRDRWB61mzak19C3bl1SeqZ7cOGkSvPoqlC9vSWwv\nQxKAEAlAKs9UVPukP2NLAN9/b0aNiQQlPCKcLmu78GrqV2lbrO2TG48cMcO8P/8cXBLPZTVOkSql\n0iil1iqlTkb+mzqacn8qpQ4rpQ4opfbE5ZhCOKq2vm2ZUjsHF9ImQbdqBffuWR2SeMz0A9MJvBrI\n8IrDcXd1f3LjpEng7g7NmlkS28uKa6rqAazXWucB1ke+j867Wus3tda+cTymEA7Jw82D/h+OouUH\nD1HHjsGQIS/+krCL2w9u02djH0pmLUnt/LWf3Hj3LsycaRp/E+Cyj88T1wRQA5gR+fMMIOGteixE\nIlI7f21CK5RhQREP9NChcPiw1SEJYMT2EVy+fZlR74/6b8qHR+bPNw33bdpYE1wcxDUBpNdaX4r8\n+TKQPppyGlinlNqrlGodx2MK4bCUUnxT+RvavXefO17u0LIlhMk0EVY6e/0sIwNG0qBQgyenfHhk\n4kTIn9+M/k1kXpgAlFLrlFKBUbyeWORSmzXuohvFUlpr/SbwAdBeKVX2OcdrrZTao5TaExwcHJtz\nEcIh+Gby5cOSTWhd6T7s3g1jxlgdklPrvq47CsXwisOf3bh/P+zaZe7+n34ySARemAC01hW11oWi\neC0DriilMgJE/hvl5OZa64uR/14FfgaKP+d4k7XWvlprX59EVp8mhK0MrjCYZW+4s7NYJvDzM9ML\nCLvbenYr84/Mp1upbmRLme3ZAuPGmZG/n35q/+BsIK5VQMuBR6MemgLLni6glEqulPJ+9DNQCQiM\n43GFcGhZUmShV5ne1Cr7Fw89I6uCZM5ou4rQEXRc3ZEsKbLQrVS3ZwtcuQI//WR6/qSOsgNkghfX\nBDAMeF8pdRKoGPkepVQmpdTKyDLpgW1KqYPALmCF1vq3OB5XCIfXuWRnkmbNSd8a3rB9u7nbFHYz\nbf809l3ax/CKw0mWJNmzBSZOhAcPoEPiXdpT6QQ8+ZSvr6/es0eGDQjntezYMmrOq8nptQXIuf8M\nHDwIefJYHZbDuxZ6jbzf5SVv2rxsbb712Z4/9+9Dtmzg6wsrVlgTZDSUUntj2t0+8QxZE8IJVc9b\nnUq5K/FBufNEeEQONAoPtzosh+e3wY9rodcYX3X8sxd/gLlz4epV6NjR/sHZkCQAIRIwpRT/q/I/\nTiUN5cdmb0BAAHz7rdVhObR9l/Yxce9E2hdrT+EMhZ8toLX5GxQsCBUrPrs9EZEEIEQCly9dPjq/\n05nPU24lpFIZ6N0bjh61OiyHFKEjaL+yPemSpWPAuwOiLrRpk6mK69gxUXb9fJwkACESAb+yfmRP\nlZ06715Fe3ubbocPHlgdlsOZfmA6Oy/sZETFEc/O9vnI8OFmyodPPrFvcPFAEoAQiUBy9+SM/WAs\nW+4fZ+nXH5jFhQcOtDoshxJ8J5iua7tSOltpPi0cTb/+vXth9Wro1AmSJrVvgPFAEoAQiUT1vNWp\nnrc6jVnM7U/qmcniAgKsDsthdF7TmVv3bzGp2iRcVDSXxqFDIWVKaNs26u2JjCQAIRKRsVXGAtC8\n/A10tmymKuj2bYujSvzWnV7HrEOz6F6qOwV8CkRd6OhRs2TnF1+YJOAAJAEIkYhkT5WdQe8OYtHF\nNazv1wTOnEnUA5ESgtCHobT5tQ250+Smd9ne0RccPhw8PeGrr+wXXDyTBCBEItOhRAeKZy5Ow+AJ\n3O3aEaZNM1MSi5cyYPMATv1ziokfTsTTzTPqQn/+CbNnQ+vWiW7O/+eRBCBEIuPq4sqPH/3I9XvX\naed7BUqUMEsR/vmn1aElOrsv7mZEwAiav9mc93K9F33BoUPNUo+dO9svODuQBCBEIvR6+tfpWbon\nM/6Yw6bBrc1EcY0by9oBsXA/7D7NlzUng1cGvqn8TfQFg4JgyhSTZLNmtV+AdiAJQIhEqneZ3hTw\nKcAnB/24PXa0mTCuf3+rw0o0Bm4ZyJHgI0yuNjn6Pv8A/v5mvd/ez2kfSKQkAQiRSHm4eTCz5kyu\n3rlKmxSboXlzGDwY1q61OrQEb9+lfQzbNow3VRPavPchnp5QpYrp5v+EwECYM8c0tGfIYEms8UkS\ngBCJWNFMRelTpg8/Hf6Jpe0rQoECZoTqX39ZHVqCFfowlMZLGuMR9gopAsawahUEB0OdOiYJnDz5\nWGE/P/D2hm5RrAfgACQBCJHI9SrTi6IZi9Jqw1eETJ8Ad+5Ao0bSHhCN7uu6czTkKGr5NJbNS0Oh\nQuYa36qVGd81dmxkwd27YelS6NIF0qSxNOb4IglAiEQuiWsSZtaaya37t2h6Yjh6wgTYvBn69bM6\ntARnddBqxu0aR92sHXg9aWVSPVX1X7YsHDmCmfGzUyfT5TORT/n8PJIAhHAABXwKMKrSKFaeXMn/\n8v5jlpAcMgSWPbNKq9MKuRtCs2XNKOhTkBGVhnH8ONy8+WSZgADIlw9YtAi2bYNBg8zjgYOSBCCE\ng2hfrD0fvfYR3dZ2Y3/vlma1qiZN4MQJq0OznNaaFsta8Pfdv/mp9k/kzJKUunXh449Nnf+DBzBz\npll186vWodC1K7zxhkmkDkwSgBAOQinF1BpT8UnuQ4MVzbgzd6bpvlirltPPFzR6x2h+OfELoyqN\n+neRl/HjTY4sVcpM7DltGvzyC+RdOQbOnjWLvri6Whx5/JI1gYVwMBvPbOS9me/R8PWGzE7RHFW5\nMtSsCQsXmtGsTibgfABlp5WlZr6aLKy3MMolHiMiIn81f/0Fr70GlSqZid8SIVkTWAgn9m7Od+lf\nvj9zDs9hfIpjMHKkuZg54SCxkLshfLzoY7Knys6U6lOiXt+Xx/Jily7w8KH5nTkBSQBCOKDeZXtT\n7bVqfL36awLqljCDxAYMgAULrA7NbsIiwmi0uBFX71xlYb2FpPR8wRTOq1aZxd5794ZXX7VPkBaT\nBCCEA3JRLsyqNYvsKbNTd2E9Lo/sZyq7mzUDJ6lW7ba2G2tPr2VC1QkUyVjk+YVv3zaDAPLnh+7d\n7RNgAiAJQAgHlcozFUs+XsL1e9eptbQBofN/gldegY8+cviZQ2ccmMGYnWP4sviXtCwSg548/fqZ\nht/Jk8HDI/4DTCAkAQjhwN5I/wazas1i54WdNN/RnYgVv0JoKFStCv/8Y3V48WLnhZ20/rU1FXJW\nYHSl0S/+wp49psdPmzZQunT8B5iASAIQwsHVKVCHYe8NY/6R+fhfXWCmNwgKMt1D79+3OjybCroW\nxEdzPyJLiiwsqLuAJK5Jnv+Fu3fNspoZMpg5/52MJAAhnEC3Ut1o8WYLBm4ZyIxUZ2H6dDNdRJMm\nEB5udXg2ceX2FSrProzWmlWfrCJtsrQv/lLXrnDsGMyYwTPzQjgBSQBCOAGlFN9X+56KuSrScnlL\nlhZJZro6LlhgGj8T8HigmLj94DbV5lbj0q1LrGi0gtfSvvbiL61YARMmmDl/KlaM/yATIEkAQjgJ\nd1d3fv74Z3wz+fLxoo9ZX/st6NULfvgBevSwOryXFvowlFrza7Hv0j4W1FtAiSwlXvylq1ehRQsz\n3cOQIfEfZALlZnUAQgj78XL3YuUnKyk/vTw15tVg7edreOf6dRgxwkx61qeP1SHGyr2we9ScX5P1\np9czveZ0qr1W7cVfCgsz02XfuAHr1ztVr5+nyROAEE4mTdI0rPl0DRm9M1Lpp8ps7lzXNIT6+ZnZ\nLxOJ+2H3qT2/NmtOreHH6j/SpHCTmH2xRw9z4Z84EQoVit8gEzhJAEI4oQxeGdjcbDNZU2Slytyq\nrO7d4L8kMGCA1eG90O0Ht6k+rzqrglYxudpkWrzVImZfnDMHRo+G9u3NoDgnJwlACCeVyTsTm5tt\nJl+6fFRfWItFXT+Epk3NoCg/vwTbMHz1zlXenfEu60+vZ0r1KbQq2ipmX9y/Hz77DMqUgTFj4jfI\nREISgBBOzCe5DxuabKBoxqLUW9KA4c1fQ7doYaqC2rdPcF1ET107RamppThy9QhLGyyN+Z3/mTNm\n8FvatGZW1CQvGB/gJCQBCOHkUidNzYamG2hYqCE9NvWmWbWHhHXtAt9/Dw0bJpjBYqtOrqLYD8W4\nFnqN9U3Wx6zBF+DKFTO98/37sHo1pE8fv4EmIpIAhBB4unnyU+2f6F++PzMPzaJkns2EDOhh7pYr\nV4aQEMtii9ARDNg8gA/nfEi2lNnY9dku3sn6Tsy+fPMmfPABXLxo+v0XKBC/wSYykgCEEIAZLNa3\nXF+W1F/CyWsnyen2HVuHtIGdO6F48cjV0u3r3I1zVJldhX6b+tH4jcYEtAzg1TQxnKr5+nWoUgUO\nH4bFi+GdGCYNJyIJQAjxhFr5a3GozSGKZCxC2QcT6elXkvC7d8wFdPlyu8SgtWby3skUmlCIgPMB\nTPxwIjNqziBZkmQx28HVq1C+vJnobf588xQgniEJQAgndP06rFwJ27eb5RCfljVlVjY02cCQCkP4\nlh0UaHKLyxlTQI0a0LmzWUU9nuw4v4Oy08vy+a+fUyxzMQ63Pcznvp9Hu5rXM86fNz19TpyAX3+F\n2rXjLdbELk4JQClVTyl1RCkVoZSKdg1KpVQVpdRxpVSQUirxjjkXwgF89x3kzGl6QrZpY6rFjx17\ntpyriys9y/TkaPuj5C9SiRz1LjKjtDd88w0RpUrC6dM2jSvwaiB1FtSh5NSSnPz7JJOrTWbdp+vI\nmTpnzHeyZYtZ6f3yZVi71jT+iuhprV/6BeQH8gKbAN9oyrgCp4BcgDtwECgQk/0XLVpUCyFsZ9s2\nrbNk0fr0afM+IkLrSZO0zpdP6/Dw53931clV+u0f39a166Oveyp93zOJvjKkt9ZhYS8dz4OwB3p+\n4Hxdblo5jT/aa4iXHrBpgL51/1bsdhQRofX//qe1m5vWefNq/ccfLx1TYgfs0TG9hse04HN38vwE\n8A6w+rH3PYGeMdmvJAAhbKtlS61Hj37ys4gIrQsW1Hr79hd/PyIiQm88s1F/8m1ZvSI3WoM+mCu5\n/n5qe73u1Dp9897N534/LDxMn/z7pJ6yb4quu6CuTjE0hcYfnfPbnHrEthE65E5I7E/q8mWt69Y1\nl7Pq1bW+fj32+3AgsUkA9pgMLjNw/rH3F4AYTNcnhLC1Gzee7QavlPnsxo0Xf18pRfkc5Sn/1WYu\nNrvAqm+6887oBRRsOZ4fi4zn03cVnllykMErAxm9M+Lp5sm9sHvcC7vHhZsXOB5ynPvhZlxBJu9M\n1C9Qn9r5a1M5d2VcVCxrpLWG2bOhY0ezpu/QodCtG7hI02ZMvTABKKXWARmi2NRba73M1gEppVoD\nrQGyZctm690L4dTefx9mzjTjux5dJ0+fhn37oGTJ2O0rc8osZO7/E3QYy/1+fWg16QeaH3FhVdVk\nzC7nyh/3j/Eg/AGebp54unmSJUUW3s/1PvnT5adAKl+W/fAGC8cr1oabdlo/P0idOoYH37bNzFy6\nebPpnTRlilnQXcROTB8VnvdCqoCESBRCQ7UuU0br99/XetYsrUeO1DpzZq3Hj7fBzk+e1Lp+fa2V\n0trdXesWLbTet8/UMT0mPFzrcuW0btBA60OHTHV9y5ZaFymi9YMHz9l/eLjWGzZoXamSqe5Jn17r\n776LUxuEIyIWVUD2eFbaDeRRSuVUSrkDDQD7dCYWQjzB0xPWrIH69U2X/hMnzGDfdu1ssPPcuU2f\n+xMnzKRrc+dCkSLmztzfHw4cgPBwNmyAa9fgp5/g9dfN5h9+gGTJYNnTdQoRERAYCP37m/1XqGAe\nV0aONI8u7duDq6sNgndOyiSMl/yyUrWAcYAPcB04oLWurJTKBPyota4aWa4q8C2mR9BUrfXgmOzf\n19dX79mz56XjE0JY6No1WLTIJILNm02dfYoUnMlYkqCURXi/ZXbInh3SpAGl+GGKwuXWTVq+fw7O\nnTOzd27ZAn//bRoqKlY0UzjXrGmyhYiSUmqv1jrabvlPlI1LAohvkgCEcBB//QUbNsC2bdxcsZXk\nF47jygtmGs2ZE8qVg7JlTeNFliz2iTWRi00CkCUhhRDxL1MmaNwYGjfG8wEUKhhG2xp/0bryWVxu\n3WDVSs3yZZrvpiUnab7s5mLv6Wl11A5PEoAQwq7c3WH1eje++CIbPb83Pf1Kl4Zx2yHpaxYH52Qk\nAQgh7C5bNtMIffu2aRrw9rY6IuckCUAIYRkvL6sjcG4yZE4IIZyUJAAhhHBSkgCEEMJJSQIQQggn\nJQlACCGclCQAIYRwUgl6KgilVDBwNo67SQeE2CCcxELO17HJ+TouW51rdq21T0wKJugEYAtKqT0x\nnRfDEcj5OjY5X8dlxblKFZAQQjgpSQBCCOGknCEBTLY6ADuT83Vscr6Oy+7n6vBtAEIIIaLmDE8A\nQgghouAwCUApVUUpdVwpFaSU6hHFdqWUGhu5/ZBSqogVcdpKDM73k8jzPKyUClBKFbYiTlt50fk+\nVq6YUipMKVXXnvHZWkzOVylVXil1QCl1RCm12d4x2koM/ltOqZT6RSl1MPJcm1sRp60opaYqpa4q\npQKj2W6/a1VMV49PyC/MWsOngFyAO3AQKPBUmarAKkABbwO/Wx13PJ9vSSB15M8fOPr5PlZuA7AS\nqGt13PH8900F/AFki3z/itVxx+O59gKGR/7sA1wD3K2OPQ7nXBYoAgRGs91u1ypHeQIoDgRprU9r\nrR8A84AaT5WpAczUxk4glVIqo70DtZEXnq/WOkBr/U/k251AYl5QNSZ/X4AvgcXAVXsGFw9icr6N\ngCVa63MAWuvEes4xOVcNeCulFOCFSQBh9g3TdrTWWzDnEB27XascJQFkBs4/9v5C5GexLZNYxPZc\nWmLuKBKrF56vUiozUAv43o5xxZeY/H1fA1IrpTYppfYqpZrYLTrbism5fgfkB/4CDgNfaa0j7BOe\nJex2rZIVwRycUupdTAIobXUs8exboLvWOsLcKDo8N6Ao8B6QFNihlNqptT5hbVjxojJwAKgAvAqs\nVUpt1VrftDasxM9REsBFIOtj77NEfhbbMolFjM5FKfUG8CPwgdb6bzvFFh9icr6+wLzIi386oKpS\nKkxrvdQ+IdpUTM73AvC31voOcEcptQUoDCS2BBCTc20ODNOmgjxIKXUGyAfssk+Idme3a5WjVAHt\nBvIopXIqpdyBBsDyp8osB5pEtrC/DdzQWl+yd6A28sLzVUplA5YAnzrAXeELz1drnVNrnUNrnQNY\nBLRLpBd/iNl/z8uA0kopN6VUMqAEcNTOcdpCTM71HOZJB6VUeiAvcNquUdqX3a5VDvEEoLUOU0p9\nAazG9CqYqrU+opRqE7l9IqZnSFUgCLiLuatIlGJ4vn2BtMCEyLviMJ1IJ9WK4fk6jJicr9b6qFLq\nN+AQEAH8qLWOslthQhbDv+1AYLpS6jCmZ0x3rXWinSFUKTUXKA+kU0pdAPoBScD+1yoZCSyEEE7K\nUaqAhBBCxJIkACGEcFKSAIQQwklJAhBCCCclCUAIIZyUJAAhhHBSkgCEEMJJSQIQQggn9X8RGAAM\nDXvH1QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4da0706d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_train, y_train = create_toy_data(sinusoidal, n=10, std=0.1)\n",
    "x = np.linspace(0, 1, 100)\n",
    "\n",
    "model = GaussianProcessRegressor(kernel=PolynomialKernel(3, 1.), beta=int(1e10))\n",
    "model.fit(x_train, y_train)\n",
    "\n",
    "y = model.predict(x)\n",
    "plt.scatter(x_train, y_train, facecolor=\"none\", edgecolor=\"b\", color=\"blue\", label=\"training\")\n",
    "plt.plot(x, sinusoidal(x), color=\"g\", label=\"sin$(2\\pi x)$\")\n",
    "plt.plot(x, y, color=\"r\", label=\"gpr\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2核函数的构造"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 特征空间映射\n",
    "1. 选择一个特征空间映射，使用该映射寻找核函数，$$k(x,x')=\\phi(x)^T\\phi(x')=\\sum_{i=1}^N\\phi_i(x)\\phi_i(x')$$\n",
    "2. 直接构造核函数,核函数对应于某个特征空间的标量积，例如\n",
    "$$\\begin {align}k(x,z)&=(x^Tz)^2=x(x_1z_1+x_2z_2)^2\\\\\n",
    "&=x_1^2z_1^2+2x_1z_1x_2z_2+x_2^2z_2^2\\\\\n",
    "&=(x_1^2,\\sqrt2 x_1x_2,x_2^2)(z_1^2,\\sqrt2 z_1z_2,z_2^2)^T\\\\\n",
    "&=\\phi(x)^T\\phi(z)\\end {align}$$\n",
    "3. 可以证明两个 kernel 函数相加、相乘后都还是 kernel；一个 kernel 乘上一个正数后也还是 kernel；一个 kernel 的指数函数也还是 kernel。 基于这些性质，可以通过简单的 kernel 构造出复杂的 kernel。\n",
    "例如高斯核：\n",
    "$$k(x,x')=exp\\bigg(-\\frac{\\lVert x-x'\\lVert^2}{2\\sigma^2}\\bigg)$$\n",
    "$$\\lVert x-x'\\lVert^2=x^Tx+(x')^Tx'-2x^Tx'$$\n",
    "$$k(x,x')=exp(-\\frac{x^Tx}{2\\sigma^2})exp(-\\frac{x^Tx'}{\\sigma^2})exp(-\\frac{(x')^Tx'}{2\\sigma^2})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 由生成模型构造核函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 给定一个生成模型$p(x)$,可以定义一个核：$k(x,x')=p(x)p(x')$，$p(x)$是一个从D维空间到1维空间的特征映射，根据核函数的性质，考虑不同概率分布的乘积的加和，可以得到更复杂的形式$$k(x,x')=\\sum_ip(x|i)p(x'|i)p(i)$$\n",
    "$p(i)$为隐变量（分布）,考虑隐马尔科夫模型，可以使用一个核函数来度量两个序列的相似度(两个观测序列都通过相同哦你没个的状态序列Z生成)。\n",
    "$$k(X,X')=\\sum_Zp(X|Z)p(X'|Z)p(Z)$$\n",
    "2. 另一种方法为Fish核,考虑一个生成式模型$p(x|\\theta)$，目标是找到一个核来度量这个生成式模型两个输入变量$x$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "常见的[kernel](http://crsouza.com/2010/03/17/kernel-functions-for-machine-learning-applications/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
